AV Timing Measurement and Correction for Digital Television

ABSTRACT

An invention for measuring, maintaining and correcting synchronization between signals which suffer varying relative delays during transmission and/or storage is shown. The present invention teaches measuring the relative delay between a plurality of signals which have suffered differing delays due to transmission, storage or other processing. The preferred embodiment of the invention includes the use of a marker which is generated in response to a second signal and combined with a first signal in a manner which ensures that the marker will not be lost in the expected processing of the first signal. Subsequently a first delayed marker is generated in response to the marker associated with or recovered from the first signal, and a second delayed marker is generated from the second signal. The first delayed marker and second delayed marker are compared to determine a measure of the relative timing or delay between said first signal and said second signal at said subsequent time.

This application is a Division of application Ser. No. 14/170,786 filedFeb. 3, 2014, which is a Division of application Ser. No. 13/347,633filed Jan. 10, 2012 and issued as U.S. Pat. No. 8,810,659 on Aug. 19,2014, which is a Division of application Ser. No. 12/471,127 filed May22, 2009 and issued as U.S. Pat. No. 8,159,610 on Apr. 17, 2012, whichis a Division of Ser. No. 10/894,746 filed Jul. 19, 2004 and issued asU.S. Pat. No. 7,710,499 on May 4, 2010 which is a Division ofapplication Ser. No. 09/545,529 filed Apr. 7, 2000 and issued as U.S.Pat. No. 6,836,295 on Dec. 28, 2004 all of which are incorporated hereinby reference as fully as if they had been set out in detail and to whichpriority is claimed. Application Ser. No. 09/545,529 is in turn aContinuation-in-part of application Ser. No. 09/119,524 filed Jul. 21,1998 and issued as U.S. Pat. No. 6,351,281 on Feb. 26, 2002 which is aDivision of application Ser. No. 08/620,126 filed Mar. 21, 1996 whichissued as U.S. Pat. No. 6,330,033 on Dec. 11, 2001 which claims benefitof Provisional Application 60/008,309 filed Dec. 7, 1995. No priority toapplication Ser. No. 09/119,524, U.S. Pat. No. 6,351,281, applicationSer. No. 08/620,126, U.S. Pat. No. 6,330,033 or Application 60/008,309is claimed, but which are incorporated herein by reference, in respectto their prior art teachings.

The examiner's attention is called to incorrectly published U.S. Pat.No. 5,847,769 which is related to the present application by virtue ofcommon application Ser. No. 08/620,126. The '769 patent was withdrawnfrom issue. Despite the fact of the patent being withdrawn from issue itwas nevertheless published by the Patent Office. Applicant brings thiswithdrawn patent to the attention of the examiner out of applicant'sduty of candor.

BACKGROUND OF THE INVENTION

The invention relates to measuring, maintaining and correctingsynchronization between two signals which suffer varying relative delaysduring transmission and/or storage, and in particular to measuring therelative delay between multiple audio signals and an associated videosignal of a television type program which is compressed via MPEG orother compression method for transmission and/or storage.

1. FIELD OF THE INVENTION

The present invention relates to the field of transmitting and storingmultiple electronic signals where synchronization of the signals is ofconcern. When such transmitting and storing are of a nature which makesthe corresponding receiving and recovering of said signals subject totiming errors resulting from differing amounts of processing delays thepresent invention is useful in measuring the relative timing errors ordelays between signals with such delay measurement being used as a meterof quality of the transmitting and storing and for maintaining orcorrection of relative delays between such signals.

2. DESCRIPTION OF RELATED PRIOR ART

It is known in the television signal transmission field to measure andcorrect audio to video timing errors by measuring the delay which avideo signal experiences and using that measurement to delay a companionaudio signal by a corresponding amount.

U.S. Pat. No. 4,313,135 by the present inventor shows to comparerelatively undelayed and delayed versions of the same video signal toprovide a delay signal responsive to the delay thereof and to couplethat delay signal to a variable audio delay to cause the audio delay todelay the companion audio signal by a corresponding amount.

U.S. Pat. Nos. 4,665,431 and 5,675,388 by the present inventor showtransmitting an audio signal as part of a video signal so that both theaudio and video signals experience the same transmission delays thusmaintaining the relative synchronization therebetween.

U.S. Reissue Pat. No. RE 33,535 corresponding to U.S. Pat. No. 4,703,355shows in the preferred embodiment to encode in the vertical interval ofa video signal, a timing signal derived from an audio signal andtransmitting the combined video signal and the audio signal. At thereceiving location the timing signal is recovered from the video signaland a new timing signal is generated from the received audio signal. Thetwo timing signals are compared at the receiving location to determinethe relative delay between the timing signal recovered from the videoand the newly generated timing signal, thus determining the relativedelay between the video and audio signals at the receive location. It isalso suggested to put a timing signal in the audio signal.

U.S. Pat. No. 5,202,761 by the present inventor shows in the preferredembodiment to encode a pulse in the vertical interval of a video signalbefore the video signal is delayed. The encoded pulse is recovered fromthe vertical interval of the delayed video signal. Various methodsresponsive to the encoded pulse or the timing thereof for the undelayedvideo and the encoded pulse recovered from the vertical interval of thedelayed video are shown which enable the determination of the delay, orthe control of a corresponding audio delay.

U.S. Pat. No. 5,530,483 by the present inventor shows determining videodelay by sampling an image of the undelayed video and sampling images,including the same image of the delayed version of the video andcomparing the samples of the undelayed image to the samples of thedelayed images until a match is found indicating that the undelayedimage in delayed form is being compared. The time lapse between thesampling of the undelayed image, and the finding of the matching delayedimage is used as a measure of video signal delay.

U.S. Pat. No. 5,572,261 by the present inventor shows a method ofdetermining the relative delay between an audio and a video signal byinspecting the video for a speaker's mouth and determining various mouthpatterns of movement which correspond to sounds which are present in theaudio signal. The time relationship between a mouth pattern whichcreates a sound and the occurrence of that sound in the audio is used asa measure of audio to video timing.

U.S. Pat. No. 5,751,368, a CIP of U.S. Pat. No. 5,530,483 shows the useof comparing samples of relatively delayed and undelayed versions ofvideo signal images for determining the delay of multiple signals.

Applicant incorporates all of the above prior art patents herein asfully as if they were set forth in their entirety for the purposes ofenabling one of ordinary skill in the art to practice the presentinvention in so far as the present invention utilizes many elementswhich are taught therein. In particular, attention is called to U.S.Pat. No. RE 33,535 and the teachings of generating a timing signal inresponse to an audio signal, and the comparison of a recovered timingsignal and a newly generated timing signal at the receiving site todetermine the relative delay therebetween.

The above cited inventions often prove to be less than completesolutions for modern television systems and others which transmit orstore a plurality of signals for various reasons including for examplethose problems recited below. In particular, the current transmission ofMPEG compressed television signals has proven to have particulardifficulty in maintaining audio to video synchronization, and the priorart has particular problems in dealing with such.

U.S. Pat. No. 4,313,135 compares relatively undelayed and delayedversions of the same video signal to provide a delay signal. This methodrequires connection between the undelayed site and the delayed site andis unsuitable for environments where the two sites are some distanceapart. For example where television programs are sent from the networkin New York to the affiliate station in Los Angeles such system isimpractical because it would require the undelayed video to be sent tothe delayed video site in Los Angeles without appreciable delay,somewhat of an oxymoron when the problem is that the transmission itselfcreates the delay which is part of the problem. A problem also occurswith large time delays such as occur with storage such as by recordingsince by definition the video is to be stored and the undelayed versionis not available upon the subsequent playback or recall of the storedvideo.

U.S. Pat. Nos. 4,665,431 and 5,675,388 show transmitting an audio signalas part of a video signal so that both the audio and video signalsexperience the same transmission delays thus maintaining the relativesynchronization therebetween. This method is expensive for multipleaudio signals, and the digital version has proven difficult to implementwhen used in conjunction with video compression such as MPEG.

U.S. Reissue Pat. No. RE 33,535 corresponding to U.S. Pat. No. 4,703,355shows in the preferred embodiment to encode a timing signal in thevertical interval of a video signal and transmitting the video signalwith the timing signal. Unfortunately many systems strip out and fail totransmit the entire vertical interval of the video signal thus causingthe timing signal to be lost. It is suggested to put a timing signal inthe audio signal, which is continuous thus reducing the probability oflosing the timing signal. Unfortunately it is difficult and expensive toput a timing signal in the audio signal in a manner which ensures thatit will be carried with the audio signal, is easy to detect, and isinaudible to the most discerning listener.

U.S. Pat. No. 5,202,761 shows to encode a pulse in the vertical intervalof a video signal before the video signal is delayed. This method alsosuffers when the vertical interval is lost.

U.S. Pat. No. 5,530,483 shows determining video delay by a method whichincludes sampling an image of the undelayed video. This method alsorequires the undelayed video, or at least the samples of the undelayedvideo, be available at the receiving location without significant delay.Like the '135 patent above this method is unsuitable for long distancetransmission or time delays resulting from storage.

U.S. Pat. No. 5,572,261 shows a method of determining the relative delaybetween an audio and a video signal by inspecting the video forparticular sound generating events such as a particular movement of aspeaker's mouth and determining various mouth patterns of movement whichcorrespond to sounds which are present in the audio signal. The timerelationship between a video event such as mouth pattern which creates asound and the occurrence of that sound in the audio is used as a measureof audio to video timing. This method requires a significant amount ofaudio and video signal processing to operate.

U.S. Pat. No. 5,751,368, a CIP of U.S. Pat. No. 5,530,483 shows the useof comparing samples of relatively delayed and undelayed versions ofvideo signal images for determining the delay of multiple signals. Likethe '483 patent the '368 patent needs for the undelayed video or atleast samples thereof to be present at the receiving location.

U.S. Pat. No. 6,330,033 and Division U.S. Pat. No. 6,351,281 show adelay tracker for a signal processing system, where the delay trackerutilizes a special code or pulse associated with the tracked signal withthe system including a pulse detector later recognizing the special codeor pulse in order to identify such signal and ascertain any delaysassociated with the signal including possible resynchronization ofassociated signals. In the preferred embodiment the invention isutilized with video and audio signals to measure or maintain lip sync.The delay tracker is associated with the video signal in a manner thatit will be carried through the processing that it is expected toreceive. In one particular example the tracker which is associated withthe video signal is generated in response to certain artifacts orcharacteristics already present in the audio signal.

The instant invention provides for improvements in the field oftransmitting and storing multiple electronic signals wheresynchronization of the signals is of concern, for example related toU.S. Pat. Nos. 6,330,033 and 6,351,281.

Attempts have been made to add various timing related signals intelevision program streams in order to maintain audio to videosynchronization. In particular in MPEG systems control signals such astime stamps are utilized. Unfortunately the inclusion of these signalsdoes not guarantee proper audio to video synchronization at the receiveside output of the system for a variety of reasons, including the factthat there are significant video delays which occur which cannot betracked by the time stamps.

BRIEF SUMMARY OF THE INVENTION

It is an object of the invention to provide a method for measuring ormaintaining the relative delay of a plurality of signals which arepassed through subsequent processing.

It is another object of the invention to provide a method of generatinga marker in response to a second signal which marker may be associatedwith a first signal in a fashion that said marker is carried with saidfirst signal through processing of said first signal.

It is still another object of the invention to provide a method ofresponding to a marker which has been associated with a first signal anda marker which is provided in response to a second signal whereby saidmarkers may be utilized to determine the relative delay between saidfirst and second signals.

It is a further object of the invention to provide a marker in responseto a signal wherein said marker indicates the occurrence of particularcharacteristics of said signal.

It is a still further object of the invention to provide a system ofmeasuring the relative delay between an audio and a video signal in atelevision system wherein the audio and video signals are subject todiffering processing which creates unequal delays in said signals.

It is yet still a further object of the invention to provide a method ofmarking a first signal which may be a video signal to allow relativedelay measurement of said first signal and a second signal which may bean audio signal after they have been processed, including use of amarker generator responsive to the second signal to generate a markerupon the occurrence of one or more particular characteristics of theaudio, associating the marker with the video signal in a fashion suchthat the marker will be carried with the video signal and not beadversely affected by the subsequent processing thereof.

It is yet still another object of the invention to provide a relativedelay measurement system for measuring the relative delay between aplurality of signals including a first signal which is a video signaland second signal which is an audio signal which signals experienceunequal delays due to processing thereof, the invention including use ofa marker generator responsive to the audio signal to generate a markerupon the occurrence of one or more particular characteristics of theaudio, associating the marker with the video signal in a fashion suchthat the marker will be carried with the video signal but not beadversely affected by the subsequent processing thereof, responding tothe marker with the video signal after the processing to generate afirst delayed marker; generating a second delayed marker in response tothe processed audio signal, comparing the relative timing of the firstand second delayed markers to determine the relative timing between theprocessed audio and processed video signal.

The preferred embodiment of the invention may be used with a televisionsignal. At the transmitting location a marker is generated in responseto the audio signal and is associated with the video signal such thatthe marker is carried with the video signal in a fashion such that itwill not be lost or adversely affected by the expected processing of thevideo signal. The audio signal and the marker associated video signalare stored, transmitted and/or processed and made available at a latertime thus becoming delayed video and audio signals. A first delayedmarker is recovered from the delayed video signal and a correspondingsecond delayed marker is generated from the delayed audio signal, withthe two delayed markers compared to determine the relative delaytherebetween. This relative delay between these markers is responsive toand is a measure of the delay between the delayed video signal anddelayed audio signal.

Somewhat simplistically stated, the preferred embodiment of theinvention operates by generation of the marker at the transmit section,which may be thought of a marking the video at the time of theoccurrence of a known event in the audio signal. The time marker isassociated with the video signal such that it is carried in time withthe video signal for all of the processing which the video signal is toexperience. After the video signal processing and any audio signalprocessing, the same event in the audio is again marked in time, and thepreviously marked time (relative to the video) is recovered or flaggedin the received video. Since it is known that the audio event and themarking of the video occurred (substantially) simultaneously at thetransmit location, the displacement between those events at the receivelocation is a measure of the audio to video timing error, or therelative delay therebetween.

Generally, the present invention teaches measuring the relative delaybetween a plurality of signals which have suffered differing delays dueto transmission, storage or other processing. The preferred embodimentof the invention includes the use of a marker which is generated inresponse to a second signal and combined with a first signal in a mannerwhich ensures that the marker will not be lost in the expectedprocessing of the first signal. Subsequently a first delayed marker isgenerated in response to the marker associated with or recovered fromthe first signal, and a second delayed marker is generated from thesecond signal. The first delayed marker and second delayed marker arecompared to determine a measure of the relative timing or delay betweensaid first signal and said second signal at said subsequent time.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a block diagram of the preferred embodiment of theinvention as used with a television audio and video signal.

FIG. 2 shows a block diagram of the marker generator 3 and 13 of thepreferred embodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

In FIG. 1 the preferred embodiment of the invention which is given byway of example, a video signal 1 and an audio signal 2 are present atwhat will be referred to as the transmit location. Either or both thevideo and audio signals may be in analog or digital, compressed oruncompressed form, the many variations and versions of which are wellknown in the art. Further. while the preferred embodiment is shown inrespect to one video and one audio signal, it will be appreciated fromthe teachings herein that the invention may be utilized and practicedwith multiple video and/or audio signals. In particular, by way ofexample the invention may be practiced with video and stereo (2channel), surround (4+channel) or 5.1 channel audio systems as arecontemplated for the new U.S. digital and HDTV transmission standards.It is also noted that the components of the invention may be implementedby analog, digital or software means or combinations thereof.

A marker generator 3 is responsive to the audio signal, and may beresponsive to the video signal as indicated by the dashed line. Inresponse to detecting the occurrence of one or more particular featureor characteristic of the audio signal generates a marker. One ofordinary skill in the art will recognize that element 44 of U.S. Pat.No. RE 33,535 may be utilized as element 3 herein. Other constructionsand operations of 3 will also be known to one of ordinary skill from thepresent teachings. The particular features, characteristics, occurrencesor other event in the audio signal which will result in the marker, willbe referred to hereinafter as occurrences and the marker in its variousforms will sometimes be referred to simply as a marker, one of ordinaryskill understanding from the context and the teachings herein thespecificity of the form or forms being referred to.

The marker from 3 is associated with the video signal 1 in a markerassociator 4. One of ordinary skill in the art will recognize thatelement 10 of the U.S. Pat. No. 6,330,033 specification can be used forelement 4 herein. Other constructions and operations of 4 will also beknown to one of ordinary skill from the present teachings. The marker ispreferred to be associated with the video signal in a fashion that themarker will not be lost, corrupted or modified beyond use by subsequentprocessing of the video signal. In particular it is preferred toassociate the marker with the video signal by including the markerwithin the active picture information of the video signal in one of themanners disclosed in detail in the U.S. Pat. No. 6,330,033specification. Consequently the marker may take on a form of activevideo, whatever form the video may be in.

Alternatively, the marker may be associated with the video signal bybeing encoded in the active video in a relatively invisible fashion byutilizing one of the various watermark techniques which are well knownin the art. Watermarking is well known as a method of encoding theownership or source of images in the image itself in an invisible, yetrecoverable fashion. In particular known watermarking techniques allowthe watermark to be recovered after the image has suffered severeprocessing of many different types. Such watermarking allows reliableand secure recovery of the marker after significant subsequentprocessing of the active portion of the video signal. By way of example,the marker of the present invention may be added to the watermark, orreplace a portion or the entirety of the watermark, or the watermarkingtechnique simply adapted for use with the marker. It is believed thatthis use of watermarking techniques to associate marker signals withvideo signals for audio to video timing purposes is novel and previouslyunknown to those in the art. Other methods of associating the markerwith the video signal will be known to those of ordinary skill in theart from the teachings herein.

The video signal with the marker is output from 4 and coupled to thevideo encoder 5. The video encoder 5 is used by way of example in thepresent description to represent that part of the subsequent videoprocessing which may take place at the transmitting side of the system.For example, the video encoder may include MPEG preprocessing andcompression circuits. Similarly, the audio 2 is coupled to an audioencoder 6 which is used by way of example in the present description torepresent the audio processing which may take place at the transmittingside of the system. For example, the audio encoder may include an MPEGcompression circuit. The compressed video and audio signals are combinedby video and audio combiner 7 and the combined signals are coupled tothe transmission channel 8.

The audio and video signals from the transmission channel 8 are coupledto a video and audio separator 9 which separates the audio and videosignal components of the transmitted signal(s). The audio and videosignals are coupled to audio decoder 11 and video decoder 10respectively, where they are decoded back into decoded audio 17 anddecoded video 16 respectively.

At the receiving side, marker separator 12 responds to the marker whichwas previously combined in the video signal by 4 to provide a firstdelayed marker to 14. The first delayed marker may be in the same formor different form as the marker which is associated with the video. Itis preferred that the marker be recovered from the video and provided asthe first delayed marker, however it is sufficient to merely detect thepresence of the marker in the video and generate a first delayed markerin response thereto. One of ordinary skill in the art will recognizethat element 40 of the U.S. Pat. No. 6,330,033 specification may beutilized for element 12 herein. Other constructions and operations of 12will also be known to one of ordinary skill from the present teachings.

Also at the receiving side, another marker generator 13, similar to 3,generates a second delayed marker in response to the same audio signaloccurrences in the receive section audio from 11 as did the markergenerator 3 on the transmit section in response to audio signal 2.Marker generator 13 may also be responsive to video in a fashion aspreviously described for 3 as shown by 19. The second delayed markergenerated by 13 need not be in the same form as the marker generated by3, but is preferred to be in the same form as the first delayed markerprovided by 12.

The first and second delayed markers from 12 and 13 are coupled to therelative timing comparison 14. The relative timing comparison isresponsive to these delayed markers to determine the timing betweencorresponding pairs thereof to determine the relative timing betweenthem. In other words the relative timing comparison 14 determines thedelay 15 of the later of the two delayed markers relative to theearlier, indicating both the magnitude of the delay and which signal ismore delayed. One of ordinary skill in the art will recognize from theteachings herein that relative timing comparison 14 may operate asdescribed with respect to element 50 of the U.S. Pat. No. 6,330,033specification. Other constructions and operations of 14 will also beknown to one of ordinary skill from the present teachings.

Since the first delayed marker from 12 experiences the delay of thevideo signal 1, and the second delayed marker from 13 experiences thedelay of the audio signal 2 in their respective paths from the input ofthe transmit section to the output of the receive section, signal 15 isa measure of the relative delay of audio 17 and video 16 at the outputof the receive section.

The relative delay 15 may be utilized for all of the uses and reasonsset forth in the U.S. Pat. No. 6,330,033 specification. In particularnote that the relative delay signal 15 is useful in itself as a measureof system quality. Relative delay signal 15 may be utilized to control adelay to delay the earlier of 16 or 17 to place the two signals intosynchronization. Relative delay signal 15 may also be utilized tocontrol a delay which is incorporated into 10 or 11 or both (orelsewhere in the system) to control the delay of the earlier of theaudio or video from 9 to maintain the two signals 16 and 17 insynchronization. Relative delay signal 15 may also be utilized for otherpurposes, for example as feedback to control the operation of encoder 5or 6 or decoder 10 or 11 to minimize or otherwise optimize delay orencoding and decoding of audio or video.

Various different embodiments of the invention herein described will beapparent to one of ordinary skill in the art from the teachings herein.As an example, the marker generator 3 may be responsive to the videosignal as shown by 18 in order to relate the marker to the video signal,for example to properly locate the marker for combination with the videosignal or to relate the particular feature(s) of the audio signal totiming of the video signal. In particular, it is desired that the markerrepresent whether or not the particular features occurred in the audiosignal during the one or more frame or field immediately prior to themarker being combined with the video and going back to the time when theimmediately previous marker was combined.

The marker is preferred to be a binary signal which indicates that oneor more of a number of particular occurrences of the audio which hastaken place during the preceding field(s), or is currently taking place.For example, an 8 bit binary signal may be utilized with differentnumbers corresponding to different occurrences or features. In thepreferred embodiment, it is preferred that the audio signal, which inthe present example is assumed to have a bandwidth of 10 Hz to 20,000 Hzbe broken up into 8 different frequency bands by bandpass filtering.Each bit of the 8 bit number corresponds to the presence of audiofrequencies within a particular band having energy within known levelsand for known time durations. For example, if no such frequencies arepresent, the binary number 0 (0000 0000) results. If the lowestfrequencies occur, the binary number 1 (0000 0001) results. If the nexthighest frequency occurs the binary number 2 (0000 0010) results. Ifboth the lowest and next highest occur a 3 results. If all frequenciesoccur the binary number 255 (1111 1111) results. The binary number isthe marker which is combined with the video.

It is important to note that by associating the marker with the videosignal in the fashion of including it in the active video portion of thesignal that the marker will not be lost when all of the sync andblanking (or line, field and other ancillary signals if in digital form)are removed from the video signal such as is done as part of the MPEGencoding process. The association of the marker directly with the imagecarried by the video signal essentially guarantees that no matter whatprocessing, stripping or modification of ancillary portions of the videosignal occurs, in either analog or digital form, or conversion ofscanning rates, or adjustment of usual video parameters such as black,brightness and chroma, that the marker will still be detectable at thereceive location.

The transmission channel 8 is utilized in the present example torepresent any common or independent use or processing of the videosignal 1 and audio signal 2 which may cause or result in unequal delayswhich lead to timing difficulties. Examples of such uses includetransmission, storage and further processing, and in particular includestorage and/or transmitting of MPEG encoded audio and video signals.

Also it may be noted that marker generators 3 and 13 may respond tovideo in other forms, or from other parts of the system, or may respondto other signals, for example a genlock reference, in order to achieveproper operation and timing of the marker generator.

It may be noted that the use of the video encoder 5, audio encoder 6 andvideo and audio combiner 7 is given by way of example, as is usual forMPEG compression and transmission systems which are commonly used intoday's television systems. The invention is not limited to the use ofsuch elements however and one of ordinary skill in the art will know howto practice the generation of the marker and the associating of themarker with the video signal in other systems from the presentteachings. The combined marker and video signal from 4 and the audiosignal 2 may very well be utilized in practicing the present inventionwithout the added elements 5-7.

It will be understood that in the present example the elements 9, 10 and11 are the receiving side elements complimentary to correspondingtransmitting side elements 7, 5 and 6 respectively. As with 5, 6 and 7,elements 9, 10 and 11 are not required to practice the invention. Inparticular, video from 4 may be coupled, via a transmission channeldirectly to element 12 and become video signal 16. Similarly, audiosignal 2 may be coupled via the same or different transmission channeldirectly to 13 and become audio signal 17.

In the situation where the transmission channel includes storage of theaudio and video signals, and storage and recovery is not performedsimultaneously, it is noted that a single marker generator 3 may performthe function of 3 upon the storing of the signals and subsequentlyperform the function of 13 upon the recovery of the stored signals.Other sharing of circuitry between storing and recovery functions mayalso be had given the assumption that both are not performedsimultaneously.

FIG. 2 shows the preferred form of the marker generator 3 and 13 of thepreferred embodiment of the invention as used with television audiosignals. Audio signal 20 which may correspond to 2 or the output of 11in FIG. 1 is coupled to a bank of 8 bandpass filters 21 a-h which areconfigured to pass only audio within a range of frequencies as is wellknown in the art. The output of each bandpass filter is coupled to acomparator 22 a-h respectively. The comparators include hysteresis orother threshold(s) and bipolar response characteristic so that if thepositive or negative half cycle of bandpassed audio out of the bandpassfilter exceeds a threshold amount set by the hysteresis, the output ofthe comparator is activated. Each comparator output is respectivelycoupled to a timing duration circuit 23 a-h. Each timing durationcircuit also receives a reset signal from the timing circuit 26. Thetiming circuit 26 provides signals to the parallel to serial converter24 in addition to the reset signal provided to the timing durationcircuits 23. Once the timing duration circuit is reset, it inspects theoutput signal from its respective comparator 22. If the output signalfrom 22 is activated for an established time duration indicating thepresence of audio frequencies within the corresponding bandpass filterrange, the timing duration circuit sets its output active and holds itactive until the next reset signal. The outputs of all of the timingduration circuits 23 are simultaneously latched into the parallel toserial circuit 24 upon command from the timing circuit 26 and shortlythereafter the reset signal to 23 is generated. Also shortly afterlatching, the bits latched into 24 are caused to be output in serialfashion as marker 25. The net effect of the circuitry is to set a bit ofthe timing signal active corresponding to each of the bandpass audiofrequencies which was present during the time period from one resetsignal to the next, which corresponds to the time period from thegeneration of one marker to the next. The timing circuit 26 isresponsive to the video signal to set the desired time period betweenmarkers, as well as to time the output of the marker 25 so that it isassociated with the video signal at the correct time. This action willensure that the marker is placed at the desired position in the videosignal.

The bandpass filters are preferred to be selected to provide frequentoutputs with the expected types of audio signals. For commercialtelevision audio signals it has been found that bandpass filters withcenter frequencies of 25, 50, 150, 400, 1000, 2500, 6000, 15000 Hz andskirts of 6 dB per octave work well. Other center frequencies andbandwidths may be chosen, and the number of filters changed, tofacilitate expected audio signal frequency content. Ideally thefrequencies would be chosen such that the lowest frequency filter has anoutput which is active or makes a change of state only once per periodof the maximum expected delay differential of the audio and videosignal. Alternatively, other audio characteristics may be relied on inthe place of, or in addition to, the detection of energy at particularfrequencies as described in respect to the preferred embodiment.Examples include, but are not limited to, impulse characteristics,amplitude characteristics, relationships between different frequencyenergies, relationships among and between different audio channels.

Another example of alternate audio characteristics which may be utilizedfor the marker is the particular audio sonic characteristics which arerelied on for the audio compression. Because these characteristics arealready detected in the compression circuitry the present invention mayshare circuitry thus resulting in lowered cost. Other sharing ofcircuitry with other functions may be possible depending on theparticular signals and environment with which the invention is used.

While it has been described to utilize the marker generator with oneaudio signal in the preferred embodiment, it will be understood thatmultiple audio signals may be accommodated, with each having acorresponding marker which is associated with the video. Alternatively aplurality of audio signals may be used to generate a lesser number oreven one marker by various techniques which include combining theplurality of audio signals before coupling to the marker generator, orby combining various markers each responsive to one or a small number ofaudio signals with the various markers being combined into a smallernumber or a single master marker.

It may be noted that many audio ICs which are used for audio graphicequalizer functions contain bandpass filters which may be adapted to usein this invention. Of course it is possible to implement the variouselements of the marker generator, as well as the rest of the invention,in analog or digital hardware, or software/hardware or combinationsthereof.

It will be noted that the present description of the preferredembodiment of the invention is given by way of example. In particularthe diagrams of the preferred embodiment are presented as block diagramsand do not show in detail circuitry and cooperation which would be knownto those of ordinary skill in the art from the teachings herein withoutundue experimentation. By way of example it is noted that where onesignal line is shown in the block diagram that multiple signals may inactuality be coupled between one block and another, and althoughseparate functional blocks are shown it will be known to make differentcombinations, arrangements and implementations in order to shareelements therebetween and reduce costs. It is also noted that variousterms used in the specification, including generator, combiner, encoder,separator, decoder and comparison, and their various tenses are intendedto have broader meaning than that ordinarily ascribed thereto withrespect to circuit elements, and are intended to cover not only thecommonly understood element but the equivalent operation or function asimplemented by other circuitry or software/hardware combinations. One ofordinary skill in the art will know to resort to various changes andmodifications to the invention as described as well the combination ofthe invention with other features functions and/or inventive concepts inorder to accommodate the use of the invention with particular forms ofsignals and otherwise to practice the invention in a fashion which isoptimized for particular application without departing from the spiritand scope of the invention as hereafter claimed.

1. An apparatus for generating a first set of markers in response to arelatively undelayed or delayed plurality of channels audio signal whichis part of a high definition television program which first set ofmarkers are intended to be utilized with a second set of markersgenerated in response to a delayed or undelayed version respectively ofthe audio signal, said apparatus including: a) an input circuitresponsive to a digital audio portion of a high definition televisionprogram, said input circuit including combining a plurality of channelsof said digital audio portion to provide at least a first digital outputsignal capable of carrying energy in a range of audio frequencies; b) aninput filter circuit comprised of at least two digital filters, eachresponsive to said first digital output signals, each having a differentamplitude response characteristic and each operative to provide adigital filter output signal having an amplitude which is responsive toone or more characteristic of said digital output signal; c) a markergenerator circuit responsive to said digital filter output signals andin response thereto generating a sequential plurality of first markerswherein the sequence of said sequential plurality of first markers isgenerated in response to the timing of frames of video of said highdefinition television program.
 2. An apparatus as in claim 1 where saiddigital audio portion of a) is made up of 5.1 channels of digital audiowhich are all combined to provide said first digital output signal. 3.An apparatus as in claim 1 where said sequential plurality of firstmarkers of element c) are time sequential in response to the timesequence of said frames of video of element c) and said first markerswhich are generated during the time period from one frame of video tothe next frame of video are associated with said next frame of video. 4.An apparatus as in claim 1 wherein said one or more characteristic ofsaid digital audio portion of element b) is one or more of: energy atparticular frequencies; impulse characteristics; amplitudecharacteristics; relationships between different frequency energies;relationships among different audio channels and/or relationshipsbetween different audio channels.
 5. An apparatus as in claim 1 whereinin element b) said at least two digital filters are each responsive tothe amplitude of said first digital output signal of element a) toprovide a different respective said digital filter output signal asdetermined by the respective said different amplitude responsecharacteristic of each said digital filter.
 6. An apparatus as in claim1 wherein in element b) two digital filters are each responsive to theamplitude of said first digital output signal of element a) the twodigital filters thereby providing a first filter output signal and asecond filter output signal as determined by the respective saiddifferent amplitude response characteristic of each said digital filterand wherein in element c) said marker generator circuit is responsive tothe amplitude of said first filter output signal and the amplitude ofsaid second filter output signal to generate said sequential pluralityof first markers.
 7. An apparatus as in claim 1 wherein in element b) afirst digital filter of said at least two digital filters is responsiveto energy of said digital audio portion and said different amplituderesponse characteristic to provide a first said digital filter outputsignal and wherein in element c) said marker generator circuit includesat least one comparator circuit which is responsive to said first saiddigital filter output in order that said markers are generated inresponse to the filtered positive and negative half cycles of saiddigital audio portion exceeding a threshold of said comparator circuit.8. An apparatus for generating a first set of markers in response to arelatively undelayed audio signal having one or a plurality of channelswhich is part of a high definition television program which first set ofmarkers are intended to be utilized with a second set of markersgenerated in response to a delayed version of the audio signal, saidapparatus including: a) an input circuit responsive to one channel or acombined plurality of channels of the digital audio portion of a highdefinition television program and providing a first digital outputsignal capable of carrying the audio energy present over a range ofaudio frequencies in said channel or said combined channels; b) an inputfilter circuit comprising at least two digital filters, each saiddigital filter having a different amplitude response characteristic ascompared to the other said digital filter(s) with each said digitalfilter being responsive to the positive and negative half cycles of saiddigital audio portion as carried by said first digital output signal toprovide a digital filter output signal having an amplitude which isresponsive to one or more characteristic of said positive and negativehalf cycles of said digital audio portion and the respective saidamplitude response characteristic for that filter; c) a marker generatorcircuitry including digital comparator circuitry and responsive to twosaid digital filter output signals corresponding to said at least two ofsaid digital filters and in response thereto said marker generatorcircuit generating a sequential plurality of first digital markerswherein said sequence is generated in further response to frames ofvideo of said high definition television program; d) with saidsequential plurality of first digital markers being associated with saidvideo of said high definition television program such that saidplurality of first digital markers will be carried with said video ofsaid high definition television program as it is delayed by furtherprocessing and becomes a delayed high definition television program witha delayed audio portion and delayed video portion.
 9. The apparatus ofclaim 8 further including: e) an second input circuit the same as a) butresponsive to said delayed audio of d) to provide a second digitaloutput signal; f) a second input filter circuit the same as b) butresponsive to said second digital output signal to provide seconddigital filter output signals; g) a marker generator circuitry the sameas c) but responsive to two said second digital filter output signalsoff) corresponding to said at least two of said digital filters of f)and generating a sequential plurality of delayed digital markers whereinsaid sequence is generated in further response to frames of delayedvideo of said delayed high definition television program; h) acomparison circuit wherein a first set of markers taken from saidsequential plurality of first digital markers of c) is compared with asecond set of markers taken from said sequential plurality of delayeddigital markers of g) to determine the advance or delay of said delayedaudio of d) relative to said delayed video of d).
 10. The apparatus ofclaim 9 wherein said first set of markers is taken from said sequentialplurality of first digital markers of d) which have been associated withsaid delayed video of said delayed high definition television programand are subsequently recovered from their association with said delayedvideo before their comparison in h).
 11. The apparatus of claim 9wherein said comparison circuit of h) includes a correlator forcorrelating said first set of markers of h) with said second set ofmarkers of h) to determine the advance or delay of said delayed audio ofd) relative to said delayed video of d).
 12. The apparatus of claim 10wherein said comparison circuit includes a correlator for correlatingsaid first set of markers which are subsequently recovered from theirassociation with said delayed video with said second set of delayedmarkers of f) to determine the advance or delay of said delayed audio ofd) relative to said delayed video of d).
 13. In a digital systemoperating with a high definition television program having video andaudio portions the audio portion having one or more channels and where asequential plurality of first markers is associated with the video ofthe high definition television program before the high definitiontelevision program is delayed by further processing to become a delayedhigh definition television program with delayed audio and delayed videoportions, an apparatus for generating delayed markers in response to thedelayed audio which delayed markers are intended to be utilized with thefirst markers which are recovered from their association with thedelayed video to determine whether the delayed audio leads or lags thedelayed video, said apparatus including: a) a delayed audio inputcircuit responsive to positive and negative half cycles of a channel orcombined plurality of channels of delayed audio of a delayed highdefinition television program and in response to said half cyclesproviding a delayed digital output signal carrying energy of saidchannel or combined plurality of audio channels over a range of audiofrequencies; b) a delayed input filter circuit comprised of at least twodigital filters, each said digital filter having a different responsecharacteristic as compared to the other said digital filter(s) each saiddigital filter operative to provide a corresponding digital filteroutput signal having an amplitude which is responsive to the respectivesaid digital filter's response characteristic and to one or morecharacteristic of said positive and negative half cycles of said delayedaudio which half cycles are carried by said delayed digital outputsignal; c) a marker generator circuit including a comparator circuitresponsive to at least a first said digital filter output signal of b)to provide a comparator output signal and in response to said comparatoroutput signal said marker generator circuit generates a sequentialplurality of delayed markers; d) a comparison circuit wherein saidsequential plurality of delayed markers of c) is in the same form as asequential plurality of first markers which were previously associatedwith the video of said delayed high definition television program saidcomparison circuit operating to compare a first set of makers taken fromsaid sequential plurality of delayed markers of c) with a second set ofmarkers taken from said sequential plurality of first markers which havebeen recovered from their association with the video of said delayedhigh definition television program.
 14. An apparatus as claimed in claim13 wherein said positive and negative half cycles of said delayed audioof a) have an amplitude and are carried by said delayed digital outputsignal of a) which delayed digital output signal carrying the positiveand negative half cycles are coupled to and filtered by one of saiddigital filters of b) to provide said first digital filter output signalof c) having an amplitude responsive to said filtered positive andnegative half cycles which said digital filter output signal is coupledto said comparator of c) and when the digital filter output signalamplitude exceeds a threshold amount the output of said comparator isactivated thereby generating said comparator output signal of c).
 15. Anapparatus as in claim 13 wherein said sequential plurality of delayedmarkers of c) are responsive to one or more characteristic of saidplurality of channels of delayed audio of element a) whichcharacteristic is one of energy at particular frequencies, impulsecharacteristics, amplitude characteristics, relationships betweendifferent frequency energies, relationships among different audiochannels and/or relationships between different audio channels, of saiddigital audio portion.
 16. An apparatus as in claim 13 wherein saidsequential plurality of delayed markers of c) are responsive to therelationship between energies of said plurality of channels of delayedaudio of element a) being present in different ranges of audiofrequencies less than said range of audio frequencies which saidplurality of audio channels is capable of carrying of element a).
 17. Anapparatus as in claim 13 wherein said delayed markers of c) represent athreshold being exceeded by an amount of energy in a range offrequencies less than said range of audio frequencies which saidplurality of audio channels of element a) is capable of carrying.
 18. Anapparatus as claimed in claim 13 wherein said delayed markers of c) areresponsive to the amplitude characteristics of said digital filteroutput signal from each of said at least two digital filters of saidinput filter circuit.
 19. An apparatus as claimed in claim 13 whereinsaid delayed markers of c) are responsive to the amplitude of each saiddigital filter output signal from each of said at least two digitalfilters of said input filter circuit exceeding a correspondingthreshold.
 20. An apparatus as claimed in claim 13 wherein two of saidat least two digital filters of said input filter bank has acorresponding digital filter output signal having an amplitude which isrepresentative of audio energy passed by the respective filter and saiddigital filter output signal amplitudes from said two digital filtersare compared to determine the relationship between different frequencyenergies of audio with said delayed markers of c) being responsive tosaid relationship.